Fast and robust camera-based motion tracking using FPGAs in microrobotics

In microhandling, the optical sensor is often the only way to obtain position information about tools and specimen. Optical microscopes are used to observe e.g. a manufacturing process. Image processing is extensively used if handling processes are fully automated. Based on the image sensor, objects are detected and classified. Additionally, the image sensor is often used as a motion tracking system for closed-loop positioning of tools (e.g. grippers or tips). There are several reliable algorithms available for motion tracking. Still, the speed and accuracy of a closed-loop positioning system is constrained by the quality of the sensor system used. In this case, the sensor system consists of the physical camera and the motion tracking system. From a control engineering point of view, the sensor system quality is defined by characteristics such as resolution and noise, but also by timing characteristics such as update-rate, latency and jitter. A closed-loop controller can perform significantly better if the sensor signal has a high update-rate and a low and predictable latency. A predictable and well known latency additionally allows for controller optimizations. High speed motion tracking is especially required in microrobotics because small objects have very low inertia and have high-speed dynamics. Motion tracking using off-the-shelf computers and image-processing software has high latencies and high jitter. The achievable update-rates are up to 100 Hz. Experiments show that the update-rate has a high jitter, correlating with the used algorithm, the image content and the system load. Fast and robust motion tracking needs higher update-rates and a predictable runtime, thus a very low jitter. Therefore, for fast and robust motion tracking, different techniques are needed. Different approaches to improving the timing performance of motion tracking are exploited. First, a real-time operating system was used to control the scheduling of tasks. This ensured that the motion tracking would not be preempted by other processes running on the computer. Secondly, a hardware-software co-design strategy was used: An FPGA-implementation of a tracking algorithm was utilized on an embedded system. Using an FPGA, a major part of the image processing can be done in parallel performing hardware blocks. Additionally, a stream mode implementation can be used. Stream processing does not operate on an image that is available in memory, but on single pixels that are sequentially streamed through the hardware. Therefore, the image can be directly processed while its data is acquired from the camera chip and, instead of written back to memory, directly